












































DECISION SCIENCE CONSORTIUM, INC 


A PERSONALIZED AND PRESCRIPTIVE DECISION AID FOR 
CHOICE FROM A DATABASE OF OPTIONS 


Marvin S. Cohen, Kathryn B. Laskey, and Martin A. Tolcott 

Decision Science Consortium, Inc. 

7700 Leesburg Pike, Suite 421 
Falls Church, Virginia 22043 
(703) 790-0510 


March 1986 


Approved for Public Release; Distribution Unlimited 


Submitted to: 

Engineering Psychology Programs 
Office of Naval Research 
800 North Quincy Street 
Arlington, Virginia 22217 

Under Contract No. N00014-83-C-0485 
Work Unit Number NR DSA-010 



DTIC 

electe 


MAR 1 11986 


V 















REPORT DOCUMENTATION PAGE 


TM RESTRICTIVE MARKINGS 


H SECURITY CLASSIFICATION AUTHORITY 


.•OCCLASSIFICATION/OOWNORAOINO SCHEDULE 


A PERFORMING ORGANIZATION REPORT NUMSiRISI 


S. OISTRlauTION/AVAILASILITY OF REPORT 

Approved for public release; 
distribution unlimited 


S. MONITORING ORGANIZATION RtPORT NUMUERlSI 


«A NAMC OF PCRFORMINO ORGANIZATION 

Decision Science 
Consorti 


k. AOORtSS (City. SNN tod Z/P Codtl 

7700 Leesburg Pike, Suite 421 
Falls Church, Virginia 22043 



>A NAMS of MONITORING ORGANIZATION 

Department of the Navy 
Office of Naval Research 


Tfc. ADDRESS (City. Suit ZIP Com 

800 N. Quincy Street 
Arlington, Virginia 22217-5000 


ll NAME OF FUNOING/SPONSORINO 
ORGANIZATION 

Department of the Navy 


Sc. ADORESS (City. IMH md Z/P Co4t\ 


•b. OFFICE SYMBOL t. PROCUREMENT INSTRUMENT IDENTIFICATION NUMSER 
lit rppHmSN/ 

Contract No. N00014-83-C-0485 


TO. SOURCE OF FUNDING NOS. 


FROGRAM 
CLEMENT NO. 



62766N 


«.personalautmorisi from a Database of Options 
Cohen, Marvin^S .; Laskey. Kathryn. B.: Tolcott. Martin. A. 


IDA TIME COVEHCO 

FROM 83/07 TO 


RF66701801 


WORK UNIT 
NO. 

NR DSA- 
010 


IS. page COUNT 

52 



COSATI COOES 


1 ** • U * J| CT TEAMS (Comtin** on ivtwrt# if nee* wry «f»rf identify I) block number/ 

Decision aids, adaptive systems, man-machine interac¬ 
tion, cognitive psychology, cognitive styles, multi- 
attribute utility theory, decision analysis, database 


IS. ASST A ACT (CmImm m wwtm If nee* wry end identify by block number! Systems 


^In many decision-making contexts there is a need for aids which cater flexibly 
to individual users in their preferred ways of organizing information and 
solving problems, but which guard against potential errors or biases inherent 
in common approaches to decision making. DSC has developed principles of per¬ 
sonalized and prescriptive decision aiding which respond to this need, and 
which are based on experimental findings and theoretical models in cognitive 
psychology. In Phase I of this project, those principles were applied to the 
development of an aid for attack submarine approach and attack. In Phase II, 
the principles have been generalized trended to the development of a 

generic personalized and prescriptive evaluation system. The system incor¬ 
porates five basic cognitive interface modules to customize the user's / 


DO. DISTRIBUTION* A VAIL ABILITY OF ABSTRACT 
UNCLASSIFISO/UNLIMITEO C9 SAME AS RPT. □ OTIC USERS □ 


DDA NAME OF AESPONSISLS INOIVIOUAL 


IDT. ASSTRACT SECURITY CLASSIFICATION 


j^J)r^^iohn^J^T^are^ 
DO FORM 1473,83 APR 


Unclassified 


life TELEPHONE NUMBER 
llttltd* Amt Costl 

(202) 696-4502 


I 7D IS OBSOLETE. 


Die. OFFICE SYMBOL 

Code 1142EP 


Unclassified 

MCURITV CLASSIFICATION OF THIS PAGE 












































19. Abstract (continued) 


interaction with the aid and to provide prescriptive guidance, as well as a set 
of interactive dialogues or guides which blend all five functions in support of 
specific user-selected decision-making strategies. The aid will be tested and 
demonstrated in the context of a personnel selection application. 
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1.0 INTRODUCTION 


1.1 Objectives 

The overall objective of this project has been to develop a prototype deci¬ 
sion aid that successfully blends two features: (1) it is personalized in 
the sense that it accommodates individual differences in beliefs, values, 
preferred methods of problem structuring and analysis, preferred methods of 
organizing information and searching a database, and variations in cogni¬ 
tive "style," ranging from intuitive to analytical; and (2) it is prescrip¬ 
tive in the sense that it attempts in a variety of ways to steer users away 
from potential biases or errors commonly found in decision making and as¬ 
sociated with preferred decision making strategies. During Phase I, a con¬ 
cept for such an aid was developed and demonstrated in the context of a 
submarine commander's decision as to when to launch an attack on an enemy 
target. This decision involves a difficult tradeoff between waiting to ob¬ 
tain a more accurate fire control solution, and increasing own vul¬ 
nerability to enemy attack. The attack planning situation was rich enough 
to allow incorporation of a wide range of personalizing options selectable 
by individual users, as well as advisory prompts calling the user's atten¬ 
tion to potential pitfalls or inconsistencies and alerting him to critical 
events. This demonstration prototype aid was described in the Phase I 
Technical Report (1982). 

The original objectives of Phase II were as follows: 

(1) Complete computer implementation of the design concept 
developed in Phase I. 

(2) Design, implement, and test enhancements of the original 
concept, by increasing the degree of personalization and the 
scope of the coverage of the advisory prompts. 

(3) Design, implement, and test a general-purpose (non-testbed 
specific) personalized decision aid. 

Early in Phase II it was decided, in conjunction with the ONR Scientific 
Officer, that the general-purpose personalized aid would take the form of a 
system for evaluating and selecting options from a large database. In or- 
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der to stimulate potential commercial interest in the development of the 
concept, it was further decided that the database subject matter for ini¬ 
tial application of the system should be a domain of widespread interest, 
and the problem domain of personnel selection (from a large database of ap¬ 
plicant resumes) was chosen. The personalized aid itself, however, includ¬ 
ing the analytic model and interactive features, was to be generally ap¬ 
plicable to any kind of option evaluation problem. Finally, in order to 
ensure that the general-purpose aid had the highest degree of personaliza¬ 
tion and prescriptive capability possible, it was decided to focus the ef¬ 
fort involved in Objective (2) upon the general-purpose aid rather than the 
submarine attack planning aid. In this way, the end product would have the 
highest probability of being broadly applicable in a wide variety of 
decision-making domains. 

1.2 Outline 

This report provides, in Section 2.0, a background summary of the research 
literature describing the nature of commonly found cognitive biases in 
decision making, and the variations in problem-solving strategies charac¬ 
terizing different users, or indeed the same user at different times. It 
thus provides a rationale for the personalizing and prescriptive features. 
Section 3.0 describes the major differences between the attack planning and 
the personnel selection decision problems. Section 4.0 describes the 
general-purpose aid, highlighting the personalizing and prescriptive 
features. Section 5.0 presents conclusions and future plans. 









2.0 BACKGROUND 


2.1 General Description of the Decision Process 

The decision-making process can be conceptualized quite generally as con¬ 
sisting of a specific set of cognitive tasks (Figure 1). First, goals or 
objectives must be known or identified (if these are not present, there is 
no motivation to decide or act). Secondly, current circumstances, insofar 
as they are relevant to the achievement of a goal, are assessed. If a dis¬ 
crepancy is perceived between goals and reality, options for action are 
generated. If more than one option is available, a choice will be made. 

This is by no means a rigid sequence: the process is usually iterative 
(for example, revising goals, reassessing the situation, or generating new 
options when the choice process fails to turn up an acceptable 
alternative); and steps may be skipped (when, for example, the appropriate 
action is known based on past experience with very similar situations). 

But the basic set of possibilities is as shown, at least in many of the 
decision contexts we have considered, and some such framework is critical, 
we believe, for identifying the specific aspects of human performance where 
personalized and prescriptive aiding may be of use. 

It is convenient to break each of these major tasks down into more special¬ 
ized cognitive subtasks. For example, situation assessment consists of 
collecting and viewing data or evidence, deriving inferences, developing 
some sense of confidence in the conclusions, and continuing, perhaps, to 
draw further higher-level inferences. Again, the steps may be iterative, 
may be combined, or may be skipped altogether by :ome decision makers in 
some situations. 

(Note that the term "evidence" is quite relative; evidence in one process 
may be the highly uncertain conclusion of a prior analysis.) 















2.2 Basis for Prescriptive Features 


During the past 10-12 years a substantial amount of research has identified 
and quantified the types of cognitive biases or errors commonly made during 
the decision process. The prescriptive features of any aid must be 
designed to prevent or counteract these types of biases, to the extent 
possible. 

Each of the cognitive subtasks identified in Figure 1 has been associated, 
at least in laboratory research, with characteristic shortcomings in 
reasoning. Thus, by placing recent findings in cognitive psychology within 
this framework, we may derive a tentative specification of the types of 
prescriptive features that would be most appropriate. 

The following summary is not exhaustive; it is meant only to touch on some 
of the issues that bear on the present work. Three important themes, 
however, emerge: (1) Unaided decision processes employ simplifying heuris¬ 
tics that at best only approximate prescriptively accepted rules (e.g., 
Bayesian probability theory); (2) a typical effect of such heuristics is 
that awareness of uncertainty is suppressed; and (3) in many instances, 
biases are a result of (otherwise successful) efforts to utilize natural 
knowledge structures and processes of reasoning. 

Assimilate Evidence. Patterns of information search in laboratory tasks 
tend to avoid stringent tests of favored hypotheses (Wason, 1960, 1981; 
Einhorn, 1980). At the same time, there is a tendency to seek confirming 
evidence of an already well-supported hypothesis, rather than take action 
or consider evidence that bears on other issues (Shaklee and Fischhoff, 
1982) . 

Infer Conclusions. A number of studies, which show that a statistical 
model of a person's judgment process can outperform (in accuracy) that 
person's own judgments, suggest that people do not effectively utilize the 
information available to them in inference tasks (Dawes, 1975; Cohen, 
1982). Other laboratory results suggest possible causes. For example, 
people tend to ignore later evidence that contradicts a favored, or 





earlier, datum and to double count redundant evidence (Schum and Martin, 
1981). Also, people commonly ignore statistical, or "base rate", data and 
overweight unique or problem-specific factors (Kahneman and Tversky, 1972). 
Both of these observations suggest the predominance in natural reasoning of 
non-statistical, causal models (Johnson, 1985). Results can be distorted, 
and overconfidence can occur, when false analogies between the system and 
the model influence conclusions. When people do attempt to make statisti¬ 
cal judgments, moreover, estimates may be biased by the ease of recall (or 
"availability") of a particular class of events in a mental sampling 
(Tversky and Kahneman, 1973). 

Assess Quality of Conclusions . A number of studies show that people con¬ 
sistently overestimate their degree of certainty regarding predicted events 
and estimated quantities, even in areas where they are (rightfully) 
regarded as experts. While there is some evidence that experts (as opposed 
to college sophomores) are less susceptible to overconfidence 
(Lichtenstein, Fischhoff, and Phillips, 1982), other research indicates 
that the difference between expert and novice is slight (Kadane and 
Lichtenstein, 1982). When inference proceeds in stages (e.g., deriving the 
probability of being hit by enemy fire from information about the range of 
a threat, which is derived from bearings data), people often simplify the 
process by acting as if conclusions at earlier stages (e.g., range) were 
known to be true, rather than merely inferred (Schum, DuCharme, and 
DePitts, 1973). Similarly, the probability of a detailed hypothesis or 
scenario is likely to be judged higher than the probabilities of its com¬ 
ponents (Tversky and Kahneman, 1983). The latter effect may arise because 
additional details increase the match between the hypothesis and the user's 
mental models or knowledge structures (Leddo, Abelson, and Gross, 1984). 

Option Generation. People segment complex options into "natural" 
components, and treat the elements as if they were independent choices, 
leading to suboptimal portfolios (Tversky and Kahneman, 1981). There is a 
tendency to formulate options in terms of immediate actions that span only 
a short timeframe rather than as long-term policies, and to overlook, as a 
result, the cumulative risk of pursuing a given course of action over a 
long period of time (Slovic, Fischhoff, and Lichtenstein, 1978). In- 




dividuals differ in the degree to which they consider future choices in 
current planning (Streufert and Streufert, 1981) and in the number of op¬ 
tions they generate (Driver and Monk, 1976). Ingrained ways of viewing a 
problem tend to hinder the generation of novel and creative solutions 
(Pitz, Sachs, and Heerboth, 1980). 

Generate Possible Outcomes of Options. In considering what might happen if 
a particular option is adopted, people are subject to biases based on their 
internal causal models, as well as biases in recall, such as a heightened 
tendency to remember salient events or events that occurred very late or 
very early in a sequence. 

Assess Uncertainty of Outcomes. Some of the biases which affect situation 
assessment may also occur when predictions are made contingent on a par¬ 
ticular option. Additional pitfalls, however, include the effects of 
"wishful thinking" (e.g., higher probability assessments for high utility 
outcomes) or overcautiousness (e.g., lower assessments for high utility 
outcomes). According to Einhorn and Hogarth (1984), the size of these ef¬ 
fects will depend on the degree to which decision makers lack confidence in 
the probability estimates. This, in turn, may depend on the degree to 
which evidence for an estimate matches the type of evidence represented in 
user knowledge structures. An additional set of biases involves distorted 
conceptions of randomness in everyday judgment, e.g., the "gambler's 
fallacy" where a sequence of similar outcomes, which are in fact 
independent, is thought to increase the likelihood of a different outcome 
on the next trial. Fallacies of this sort may be inevitable by-products of 
powerful top-down or expectancy-driven processes of pattern recognition 
(Lopes, 1982). 

Assess Value of Outcomes . Decision makers do not typically consider all 
the potential outcomes of an action together. Rather, outcomes are grouped 
into "mental accounts" corresponding to natural objects or causal 
relations, and choices may depend critically on the particular grouping 
that is adopted (Kahneman and Tversky, 1982). An additional cognitive 
simplification is achieved by representing an outcome in causally relevant 
terms, by the difference it would make relative to some reference point. 
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Decisions may be significantly affected by the choice of reference levels, 
since the same outcome may be regarded as a gain or as a loss. For 
example, the outcome of a defensive tactic may be encoded as 400 men saved 
(relative to the number who would have died had nothing been done) or as 
200 men lost (relative to the status quo). An important finding by Kah- 
neman and Tversky (1979) is that decision makers are more likely to take 
risks when outcomes are represented as losses than when they are repre¬ 
sented as gains. 

Select an Option. Heuristic procedures may be adopted which reduce the 
cognitive effort that would be required in a thorough consideration of 
every option. Such heuristics have implications for the way decision 
makers search information. In Elimination by Aspects (Tversky, 1972), for 
example, search is organized by evaluative attributes. Attributes are con¬ 
sidered serially in order of importance; options falling below a cut-point 
on an attribute are eliminated at each stage, and not considered further. 

In this strategy, an option might be eliminated for missing a cut-point on 
one dimension even though it scores very highly on other dimensions. 
Tradeoffs, or compensatory relations among dimensions are thus not 
considered. In another heuristic strategy, called "satisficing" (Simon, 
1957; Svenson, 1979), information search is organized by options. The 
decision maker considers a sequence of options until he finds one that 
clears the cut-points he has selected on relevant attributes. Here again 
compensatory relationships are ignored. Payne (1981) has suggested that 
these information search strategies may correspond to the way decision 
makers organize knowledge. 

2.3 Basis for Personalized Features 

How are the users of decision aids likely to differ in their approaches to 
decision making and problem solving? What are the consequences of such 
differences for success in task performance? And how should aids be per¬ 
sonalized so as to enhance both user acceptability and quality of 
performance? 
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We consider, briefly, two general ways in which decision makers have been 
thought to differ from one another: 

• in the parameters and structure of a prescriptive model based 
on their personal beliefs and preferences; and 

• in the heuristic strategies, decision processes, and cognitive 
styles which they adopt in problem-solving. 

The interplay of findings from these areas helps define the potentialities 
and limitations of personalized decision aiding. 

2.3.1 Individual prescriptive decision models . Ironically, a driving 
force in the evolution of prescriptive theories of decision making has been 
the need to accommodate individual differences. An objective rule for bet¬ 
ting in games of chance, maximization of expected value, applies only where 
probabilities of outcomes can be mathematically defined (as in rolling 
dice) and where the desirability of outcomes is physically measurable 
(e.g., by money). Generalizations of this basic rule to situations where 
those conditions do not hold have led to the modern technique of decision 
analysis (cf., Edwards, 1954, 1961; Raiffa, 1968; Brown, Kahr, and 
Peterson, 1974). Von Neumann and Morgenstern (1947) formalized the notion 
of a subjective dimension of value, i.e., utility, and extended it to in¬ 
dividual preferences among probabilistic states of affairs. De Finetti 
(1937/1964) and Savage (1954) developed formal systems for the quantifica¬ 
tion of an individual's "degree of belief", or subjective probability, 
about uncertain propositions, and developed axiomatic justifications for 
the merging of utilities and subjective probabilities into a new prescrip¬ 
tive rule, maximization of subjectively expected utility. More recently, 
rigorous techniques have been developed for combining subjective 
preferences with respect to individual components of value into a single 
multiattribute utility measure (e.g., Keeney and Raiffa, 1976). 

The prescriptive force of decision analysis, in this form, is not to dic¬ 
tate to an individual in any absolute sense what he "ought" to do o^ 
believe. Rather, it indicates what choices and beliefs are logically 









consistent with other preferences and beliefs which he chooses to accept 
(cf., French, 1979). 

These elements of personalization are by no means shared by all prescrip¬ 
tive approaches. Techniques in operations research (e.g., cost/benefit 
analysis) commonly purport to be "objective" and "value free" (Watson, 

1981). The approach to decision analysis described above, however, has two 
important implications for personalized aids: 

(1) Decision-analytic aids do not address only the part of a problem that 
can be objectively measured. Actual decisions nearly always involve a num¬ 
ber of "soft factors" (e.g., uncertainty about the intentions of a business 
competitor or of a military foe; the relative importance of different 
objectives, like money and prestige). The decision maker's own experience 
may be the only source of relevant information in these matters, while an 
exclusively "factual" approach could be fatally incomplete. Aids which 
combine subjective and objective inputs must accommodate individual dif¬ 
ferences among users in assessments of uncertain states of affairs, at¬ 
titudes toward risk, and tradeoffs among competing objectives. 

(2) The second point is equally important, though far less widely 
recognized. Just as it does not prescribe inputs, decision theory 
constrains, but does not dictate problem structure. Typically, there is 
more than one way to express the probability of a hypothesis in terms of 
probabilities for other propositions; and there are multiple decompositions 
of the utility of an option into preferences for separate attributes. A 
good structure for a particular decision maker breaks the problem down into 
components about which that decision maker has either objective data or 
personal experience. Individuals might benefit differently from different 
analyses of the same problem. 

In particular, it has been suggested that experts differ from novices in 
their capability to individually recognize a very large number of different 
problem situations (De Groot, 1965; Chase and Simon, 1973). Klein (1980) 
argues that experts tend to reason holistically, by analogy with previous 
similar experiences, rather than by explicit analysis and computation. 
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Klein warns that imposition of analytical models may actually impair expert 
performance. In terms of decision theory, however, this distinction be¬ 
tween experts and novices is accommodated by the notion of personalized 
problem structures. The expert might produce quite creditable holistic 
judgments of problem components which he has "seen before" but which a less 
experienced individual would need to analyze into more familiar elements. 
(Nonetheless, experts too are subject to error--particularly when a problem 
which appears familiar has novel aspects; cf., Sage, 1981. Experts may 
benefit from analysis of such novel components.) The implication is that 
if decision aids are to exploit the capabilities of each potential user, a 
variety of models, with different functions and at different levels of 
aggregation, should be made available (cf., Strub and Levit, 1974). 


2.3.2 Individual strategies in inference and choice . Prescriptive deci¬ 
sion theory does not provide a description of actual performance, either in 
probabilistic reasoning or in the evaluation of actions (cf., Einhorn and 


Hogarth, 1981). Recent research in cognitive psychology has shed light on 


the internal processes and structures which people employ in such tasks, 


and how they differ. 


One line of research has explored the strategies people use in choosing 
among actions. Prescriptive theory requires that a single score for each 
option (its expected utility) be derived, which integrates all the avail¬ 
able information about that option: i.e., its score on each of a set of 
attributes, or the probabilities and utilities of its possible outcomes. 
Several descriptive models of choice behavior have been proposed, however, 
which involve more partial samplings of the available data (e.g., Payne, 
1973; Svenson, 1979). 


In Tversky's (1972) Elimination-by-Aspects (EBA), for example, (as 
described in Section 2.2 above), the decision maker sequentially considers 
each attribute, establishes a threshold, and eliminates all options that do 
not score at or above the threshold on that attribute. In the decision 
strategy called "satisficing" (Simon, 1957; Svenson, 1979), the decision 
maker adopts a conjunctive criterion involving cutoffs on one or more 
dimensions, and compares successive options to the criterion until he finds 
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one that is acceptable, whereupon he stops. These different decision 
strategies have different implications for the order in which people elect 
to receive information (Payne, 1973, 1976). Some strategies imply a search 
organized by options, others a search organized by attributes. 

Individual decision makers vary in the decision strategies which are 
reflected in their information-seeking behavior and in their verbal 
protocols (Payne, 1976; Russo and Dosher, 1981). But little work has been 
done to discover whether these individual differences are consistent across 
time and tasks (Svenson, 1979); instead, emphasis has been on the role of 
task variables. For example, when there are a large number of choice 
options, decision makers tend to select routines like EBA which quickly 
eliminate some options by more approximate methods. They may then switch 
over to routines which integrate all the available information about the 
remaining options (Payne, 1976; Wright and Barbour, 1977). 

Cognitive style has been regarded as a relatively invariant, abstract fea¬ 
ture of a decision maker's approach to information across a variety of 
tasks (cf., Sage, 1981; Libby and Lewis, 1977). Perhaps the most common 
differentiation made in this literature is represented by a related cluster 
of distinctions between "analytic" and "heuristic" (Huysman, 1970; Mock, 
Estrin, and Vasarhelyi, 1972), "abstract" and "concrete" (Schroder, Driver, 
and Streufert, 1967; Sage, 1981), "systematic" and "intuitive" (Bariff and 
Lusk, 1977; McKenney and Keen, 1974), and "scientific" and "managerial" 
decision makers. The common thread is a distinction between preference for 
formal, explicit analysis, breaking a problem down into elements, and an 
approach based on global intuition, trial and error, or "common sense". 

Unfortunately, there is little evidence establishing a relationship between 
these categories (based on self-descriptions) and actual information¬ 
seeking behavior (Zmud, 1979; Keen, undated). It has been found that sys- 
tematics generally take more time and do better in decision problems than 
heuristics (e.g., Mock et al., 1972). Other results, however, have been 
inconsistent, showing that systematics prefer more information or less in¬ 
formation and prefer either aggregated or raw data as compared to heuris¬ 
tics (cf., Libby and Lewis, 1977; Zmud, 1979). McKenney (quoted in Mock et 
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al., 1972) states that the propensity to be analytical Increases with task 
! familiarity; Klein (1980) and Sage (1981) suggest that experts will be more 

I 1 ® intuitive. 


A second problem in this literature is the failure to validate the claim 
that cognitive styles are task invariant. Studies which have attempted to 
do so have produced disappointing results (cf., Libby and Lewis, 1977), and 
recent reviews (Libby and Lewis, 1977; Sage, 1981) have shifted emphasis 
toward the influence of task features on decision styles adopted by the 
same individual at different times. Indeed, Hammond, et al., (1984) have 
shown that not only does the nature of the task influence decision style, 
but that particular ways of presenting task-related information can cause a 
sh^ft toward analytic or intuitive methods. 

In a few cases, "cognitive styles" have been defined in relation to actual 
cognitive behavior. Thus, Driver and Mock (1976) defined four styles by 
reference to two fairly specific processing dimensions: amount of informa¬ 
tion used and degree of focus. The latter refers to a tendency to consider 
only one solution, model, or option versus a tendency to entertain multiple 
possibilities. Streufert and Streufert (1981a) present criteria for 
"integrative" decision-making styles in terms of the number of, and length 
of time -'tween, information requests and decisions based upon them. 
Streufert and Streufert (1981b) report that integrative decision making 
decreases with decision urgency, but is an inverted-U-shaped function of 
the amount of information available. 


Implications for a Personalized and Prescriptive Aid 


Descriptive work on human inference and decision processes has implications 
for both the personal and prescriptive aspects of decision aiding. 


2.4.1 Personalization and efficient flexibility . "Flexibility" in and of 
itself is not a sufficient objective in system design. It is possible to 
make each of a vast number of logically possible information acquisition 
strategies equally easy, by allowing the user to indicate what he wants 
item-by-item. But such a system does not really facilitate the selection 
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of strategies as such; to deal explicitly with all possible search orders 
would be beyond the time and capabilities of both user and device. The ob¬ 
jective of personalization is to delimit the subset of strategies which an 
individual is most likely to prefer. Decision aids may then be tuned to 
facilitate explicit selection from this smaller group of strategies, while 
still affording the general "flexibility" of an arbitrary item-by-item 
search sequence. Such aids are efficiently flexible in their responsive¬ 
ness to likely user needs. 

The most natural way to acquire and process information can vary as a func¬ 
tion of the individual and the task. Several such forms of variation seem 
to occur frequently enough in performance to justify an aid design which 
facilitates their employment: 

• search organized by options or by attributes, 

• decision rules based on cutoffs or tradeoffs, 

• level of aggregation of information. 

In addition, it seems desirable that an aid facilitate differences 
involving: 

• focus on one or many options, 

• desired amount of information, and 

• time into the future over which planning takes place. 

There is little evidence that particular individuals are consistent across 
tasks in these preferences, and some indication that they are not. In the 
case of gross categories like "intuitive" and "analytic", moreover, there 
is no reliable mapping of traits onto system design features and certainly 
no indication of how different traits interact (cf., Huber, 1982). 

2.4.2 Prescriptive aiding . The danger inherent in complete flexibility 
for the decision maker is, of course, the high likelihood that one or more 
of the common cognitive biases described in Section 2.2 will result. As a 
safeguard against this, two types of prescriptive aids may be introduced: 
channeling and advisory prompting. The difference between them is largely 
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one of tactics. Channeling is implicit and proactive, in that it en¬ 
courages users, in advance, to adopt variants of their own preferred 
strategies which are less susceptible to biases and fallacies of judgment, 
by structuring the problem in such a way that those variants become natural 
and simple to execute. By contrast, advisory prompting is explicit and 
reactive, in that the system monitors the tasks performed by the human, 
identifies steps taken that are likely to lead to error, and prompts for 
the addition of procedures that mesh with the preferred strategy but would 
minimize the possibility of error; it also monitors tasks performed by the 
computer and prompts where a human contribution might improve results. 

Thus, in advisory prompting the computer senses weaknesses in a line of 
reasoning, whether its own or the user's, and offers help. Both channeling 
and advisory prompting may be viewed as examples of low-level expert 
systems. 

Some examples of how the prescriptive aiding techniques of channeling and 
advisory prompting can be blended into a personalized system are given 
below. 

While users should be able to organize displays around a variety of mean¬ 
ingful user-designated objects, the aid should facilitate the use of 
decision-related objects for this purpose. For example, channeling can 
facilitate clustering of options by their performance on a selected evalua¬ 
tive criterion. When an intermediate result or conclusion is uncertain, 
the sources of its uncertainty should be explicitly indicated. Evidence 
for a result should be available for display along with the result. In¬ 
ferential relationships in the database can be "mapped" by menus, which 
permit tracing a process of reasoning from its sources of evidence to its 
final conclusion. 

Research in cognitive psychology suggests that humans tend to seek addi¬ 
tional confirming evidence for a favored hypothesis. An advisory prompt 
might monitor a user's pattern of information requests, examine its own 
model of the problem in order to draw inferences about the hypotheses the 
user has in mind, and prompt the user if evidence or hypotheses exist which 
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the user has failed to consider but which may have an impact on his 
conclusions. 

Humans often find it difficult to assess the overall credibility of a con¬ 
clusion based on several steps of reasoning; they simplify by ignoring the 
uncertainty at early stages. Prompts might warn users, when they appear to 
be acting as if a particular hypothesis were known to be true, that a num¬ 
ber of stages of uncertainty must be kept in mind. The same type of cau¬ 
tion might be appropriate when a compound, or conjunctive, hypothesis is 
being considered. 

The user might be notified when two information sources, both of which are 
regarded as credible, have contradicted one another. He might then choose 
to readjust one or both credibility assessments downward. An advisory 
prompt might notify him on future occasions when either of the (partially) 
discredited sources is involved in an important conclusion. 

While the aid should permit user adjustment of any meaningful values 
employed in the database, channeling should selectively facilitate adjust¬ 
ment of values about which users are likely to have information not avail¬ 
able to the computer. Values to be adjusted could be decomposed by chan¬ 
neling into parameters about which users are likely to have reliable 
intuitions. Automatically computed values could be displayed as a 
reference, so users can focus on the appropriate direction and magnitude of 
the adjustment (based on the new evidence) and not have to integrate all 
the evidence to come up with an absolute value. 

Humans tend to combine evidence by a process that is more like averaging 
than like proper Bayesian inference. When adjustments fit an averaging 
pattern, advisory prompts might remind subjects to consider what conclusion 
a new bit of evidence favors, before performing an adjustment. 

Users could be prompted when information they possess may be of significant 
value, i.e., when (1) there is incompleteness of evidence or a conflict 
among lines of reasoning in the computer model of the problem; (2) the user 
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has potential access to relevant information; and (3) the result is 
expected to have an impact on choices among actions and ultimate payoffs. 

Channeling could facilitate relatively long time horizons for planning 
(e.g., by displaying appropriate scaling). Simultaneous specification of 
all components of a complex option could be facilitated by channeling. 
Displays should permit generation and simultaneous comparison of multiple 
options. Channeling should facilitate generation of options which include 
future choices or contingencies. 

Short-range planning might be more appropriate in some situations (e.g., 
where feedback is continuous and mistakes can be easily and quickly 
corrected), while long-range planning would be more suitable in others 
(e.g., where a risk appears small unless it is considered cumulatively over 
the long run). Advisory prompts might recommend that the user consider a 
shift in the time horizon under appropriate circumstances. 

Users should be prompted if they have generated and evaluated a complex op¬ 
tion piece-by-piece and if overall optimality would be significantly im¬ 
proved by considering the option as a whole. 

The user should be prompted if only one option has been considered, but 
another option exists which is superior on at least one dimension. 

The user should be prompted if contingency plans have not been incorporated 
in an option, but significant new information is likely to become available 
during its execution. 

Channeling could draw the user's attention to tradeoffs between different 
evaluative dimensions by displaying scores for an option on more than one 
dimension concurrently (e.g., costs and benefits). The aid's action recom¬ 
mendations should be explained by itemizing how options differ on all sig¬ 
nificant dimensions. Channeling should encode and display outcomes in 
terms of more than one reference point (e.g., assets lost, assets saved). 


Humans tend to employ simplified choice schemes that disregard tradeoffs. 

An advisory prompt might notify the user when he has eliminated an option 
because it fails to achieve a specified level on a favored evaluative 
dimension, if that option has significant advantages on other dimensions. 
The user might be told how much stretching of his specified criterion is 
required to readmit the rejected option. 

An advisory prompt should occur when a user entertains an option which is 
dominated (inferior or tied on all dimensions) by other options. 

While users should be free to designate any item or variable in the 
database as a criterion for alerting, alerts should also occur on a 
prescriptive basis. Users should be prompted when events occur or facts 
are learned which have high impact within an appropriate prescriptive 
model, e.g., which disconfirm previously well-supported inferential 
hypotheses or which significantly affect choices among actions. 

The distinction between channeling and advisory prompting is based to some 
extent on the source of the potential bias or error. Using the terminology 
of behavioral decision theory, if "base rate data" (i.e., a body of re¬ 
search findings) suggests that errors commonly result from certain ways of 
representing a problem, organizing the variables, assessing uncertainty, 
evaluating outcomes and making choices, channeling can be built into the 
aid to increase the chances that the user will adopt amended versions of 
these procedures that avoid all or most of the errors. On the other hand, 
if "individuating data" (i.e., the actual procedures and judgments of the 
specific user) appear to be moving the user into less-than-optimal choices, 
advisory prompting can explicitly point this out and suggest alternatives 
that deviate minimally from the user's originally preferred strategy 
(again, preserving the user's freedom to ignore the advice). In this way, 
the flexibility provided by personalization of an aid can be tempered by 
the prescriptive techniques of channeling and prompting without imposing 
undesired constraints on the user. 
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3.0 INTRODUCTION TO NEW DECISION MODEL 


As indicated in Section 1.1, the prototype aid serving as a context for 
this work was changed from one dealing with a submarine commander’s deci¬ 
sion about when to launch an attack on an enemy target, to one dealing with 
a personnel manager’s selection of a new hire from a number of applicants. 
Aside from the obvious difference between the two, namely, that the first 
deals with military tactics while the second deals with a broader civilian 
(as well as military support) decision situation, there are other, more 
fundamental differences between the two types of decisions that should be 
pointed out before the new system is described in detail. 

3.1 Degree of Model Generality 

The most significant difference is that the new decision problem is one 
that lends itself to a much more generic model, applicable to a wider 
variety of situations. The submarine attack model is typical of military 
tactical situations in which the decision maker is faced with a clear 
tradeoff between firing early or waiting for more information (which may 
increase the probability of his success but simultaneously decrease his own 
survival probability). This type of tradeoff occurs in many military tac¬ 
tical situations, but the temporal pace of the action and the factors that 
must be built into the model vary considerably. These tactical models must 
be largely tailored to specific situations, hence their generic features 
are severely limited. 

The personnel selection problem, on the other hand, is typical of a wide 
variety of decisions that involve multiple evaluative criteria or 
objectives. Thus, if properly constructed, the system can be made ap¬ 
plicable to other decisions such as choices among political candidates, 
policy options, investment portfolios, R&D programs, military plans, and 
many others. This generality can be accomplished by keeping the modular 
process programs entirely separate from the database. Thus, there can be 
modules that allow the user to build his own preference model, to ac¬ 
complish word processing functions while building the model, to search the 
database in various ways, to organize the data in various ways for analysis 









and display, to enter new data or be alerted when new data are entered by 
someone else, and to provide channeling and advisory prompting prescriptive 
aids--and these modular programs can operate on any type of data that are 
germane to the specific decision problem. 

3.2 Real Time and Stress 

Another important difference between the two decision situations is that in 
the submarine attack decision, real time plays a central role in the way 
the problem develops, and in the decision itself, and contributes to the 
build-up of stress in the situation. In the personnel selection decision 
on the other hand, although there may be a real time deadline for the 
decision, the process itself is largely self-paced, under the control of 
the decision maker, and stress plays a negligible role in the situation 
(although there may be stressful components in other applications of this 
generic model). In the submarine attack problem, the firing decision 
evolves over a period of time during which (in most cases) more and more 
information becomes available, and information seeking is always an impor¬ 
tant option to be considered. In the personnel selection decision, al¬ 
though it is sometimes possible to obtain additional information, the em¬ 
phasis is on evaluating existing data in a variety of ways in order to 
select an option. 

In the submarine case, once an action has been selected and performed, the 
problem essentially is re-set and may begin again later with a new target. 
In the personnel selection case, the problem often continues after a choice 
is made, since multiple choices are typically possible. Here, subsequent 
choices may be made from among fewer options, or new options (and new data) 
may be introduced into the decision situation. Thus, in this respect the 
personnel selection decision offers a richer set of conditions under which 
the decision process may be examined. 

3.3 Inference vs Choice 

Although it is often difficult to separate the two components of inference 
and choice in real-life decisions, there are certain key distinctions be- 
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tween them. Inference is concerned mainly with assessment of evidence re¬ 
lated to the relative likelihoods of various hypotheses being true, and the 
judgments being made are usually in the form of probabilities. Although 
these probabilities usually enter into the choice process, the distinctive 
feature of choice is the assessment of preferences regarding the various 
options available, or the predicted consequences or outcomes of these 
options. In the submarine attack decision, the more significant component 
was that of inference (about enemy position and intent), while in the per¬ 
sonnel selection decision, the more crucial component is that of preference 
assessment. 


3.4 Availability of Objective Data 


In the submarine context, there is little or no opportunity for the user to 
assess and adjust the validity of his decision model by reference to on¬ 
going results of applying the model. The personnel selection decision, on 
the other hand, is typically much richer in terms of its database of on¬ 
going results. The database can include data on the characteristics and 
performance of current and past employees (in addition to applicant 
characteristics), so that the user can in fact test his model against pre¬ 
vious data if he desires. As a result of this feature, the personnel 
selection application offers a large set of opportunities for individual 
variations in data organization and display. Users may change the features 
being considered, their importance, the sequence in which they are 
examined, and the way they are displayed, as well as the level of detail at 
which the analysis is conducted. As pointed out in Section 2.4, safeguards 
in the form of prescriptive aids must be provided to minimize the judgmen¬ 
tal biases that could emerge as a result of this flexibility, but the over¬ 
all aims of the project are well served in the context of the new applica¬ 
tion area. 












4.0 CURRENT STATE OF SYSTEM DEVELOPMENT 
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4.1 High Level System Design 

The conceptual design of the personalized evaluation system has been or¬ 
ganized around two closely related concerns: (1) Users of a large database 
of options may differ in the extent to which they know what they want or do 
not want, in the naturalness with which they make holistic vs. analytic 
judgments of preference, in the decision rules and information search 
strategies they favor, and in the extent to which they seek to validate 
choices by examining low-level evidence. (2) In the very experience of ex¬ 
amining and evaluating options, user preferences may grow increasingly 
determinate, with a corresponding shift in decision and information search 
strategies. As a consequence the system is designed to be personalized , 
that is, to conform to a user's preferred cognitive style. The user of 
such a personalized aid benefits from not having to force his/her thinking 
into an uncomfortable mold, but at the cost of relying on a strategy that, 
although familiar and comfortable, may be suboptimal. To guard against 
this risk, the aid is also designed to be prescriptive , to warn the user of 
information that may have been ignored, or errors in judgment that may have 
occurred. 

The system is built upon a set of elementary modules, consisting of a 
knowledge module, four cognitive interface modules that process and make 
changes in the information stored in the knowledge module, and a fifth cog¬ 
nitive interface module whose function is to alert the user when actions 
taken via the other modules may differ significantly from some normative 
benchmark. 

The experienced user can access these modules directly, via a user-friendly 
interface that features menu-driven, mouse-controlled graphical displays. 
Users may also wish to make use of guides, whose purpose is to step the 
user through the elementary modules in accordance with one of a set of 
available strategies corresponding to different cognitive styles. 
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4.1.1 Overview of elementary modules . A set of six basic modules, inter¬ 
connected as shown in Figure 2, form the basis of a generic decision aid. 
Depending on specific context, the importance attached to the functions of 
the modules may vary. Our focus in this report is on an evaluation aid 
(specifically, for personnel selection). This overview describes the 
generic functions of each module; the next section describes how each 
module is implemented in the specific context of personnel selection, and 
gives examples. 


1. Knowledge Module - This module encodes the system's knowledge about 
the decision context. Knowledge includes specific information about 
the decision problem (e.g., options and facts about options), general 
knowledge about the problem domain (e.g., preferences among evalua¬ 
tive criteria), procedural knowledge about how the decision process 
is to be carried out, and knowledge about the current status of the 
decision process. This knowledge base serves as input for, and is 
altered by, the cognitive interface modules. 

2. Adjust Module - This cognitive interface module allows the user to 
add to or change the system's database of specific problem knowledge 
and general knowledge about the problem domain. The user can also 
create or alter procedural knowledge, i.e., knowledge about how the 
database is to be manipulated. Inputs may be specified in different 
ways and at varying levels of ’’fuzziness," depending on the user's 
preference. 

3. Decide Module - This cognitive interface module allows the user to 
apply the system's procedural knowledge to the system's database. 

The user may specify one of several decision making strategies or 
models (e.g., multiattribute utility evaluation, elimination by 
aspects), corresponding to different cognitive styles, for database 
manipulation and choice. Evaluation proceeds to the degree permitted 
by the level of specificity of user inputs, and whatever implications 
can be drawn are displayed. 

4. Select Module - Using this cognitive interface module, the user may 
select a subproblem on which to focus, a subset of information to be 
displayed, or a subcategory of the database to be adjusted. 

5. Alert Module - This cognitive interface module prompts the user when 
events occur or facts are learned which may cause significant changes 
in user decisions. This function is most important in real-time 
decision aids, when rapid assimilation and incorporation of incoming 
information is essential. 

6. Advisory Module - This cognitive interface module prompts the user 
when he or she appears to be using a strategy or a user-computer task 
allocation scheme which may be suboptimal according to some normative 
benchmark. 
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Figure 2: Basic Modules of Personalized Decision Aid 
(as applied to Personnel Selection Aid) 













4.1.2 The Guides . A guide, as Its name Implies, has the function of 
directing the user through the basic functions in accordance with a given 
decision strategy. Thus, each guide is tailored to a particular strategy, 
and the cognitive style of the user determines which guide or guides (s)he 
chooses to invoke. The guides will in general be adaptive, enabling the 
user to observe the implications of certain inputs and, if necessary, cycle 
back to change them. Each strategy is subject to its own set of possible 
biases, and an important function of the guide is to inform the user when 
the possibility of normative violations occurs. 

4.2 Prototype Personnel Evaluation System: Elementary Modules 

As discussed in Section 3.0, the personnel selection context has several 
distinguishing characteristics which affect the design of a decision aid. 

The aid is focused on choice as distinguished from inference. In addition, 
time stress and the incorporation of uncertainty are of secondary 
importance, while data organization and display are of prime importance. 

In this section, we discuss both the elementary modules and the guides as 
they apply to the aid under development. The aid is generic in that it can 
be used in any problem domain sharing the above characteristics. 

4.2.1 Knowledge module . The knowledge module consists of three 
components. (1) First is an option database of context-specific knowledge. 
In the case of personnel selection, we would have a database of individuals 
(job applicants) and their relevant characteristics (salary demand, years 
of experience, etc.). This part of the knowledge module is, in fact, the 
only aspect of the system that is specific to personnel selection as dis¬ 
tinct from similar choice contexts. (2) The second component of the 
knowledge module is a preference model or models, and an associated set of 
decision strategies, which capture the user's knowledge of his own 
preferences in the problem domain and his or her preferred method for using 
those preferences in the choice process. Preference models may include 
numerical weights or attributes, intervals or ratios of weights, rank or¬ 
dering of weights, cutoffs, or direct evaluations of options. Thus, 
preferences may be specified by the user with varying degrees of 
"fuzziness" and may be incomplete in some or many respects. User-selected 
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decision strategies may be based on multiattribute utility, elimination by 
aspects, satisficing, or dominance, and are the means by which options from 
the database are retained or rejected. (3) The third component of the 
knowledge base is information regarding the current status of the system. 

For example, the knowledge module will keep track of which options in the 
database are "live" and which have been eliminated from consideration (via 
application of a preference model by use of the Decide module). In 
addition, the user may have indicated that, although the model is specified 
in terms of a large number of attributes, (s)he wishes to evaluate options 
based only on some subset of attributes. 

4.2.2 Adjust module . This module, which acts on the first two components 
of the database, has two major functions. The first function is to allow 
the user to make changes in the option database as more information is ac¬ 
quired (e.g., a candidate is no longer available, the performance of an op¬ 
tion on an evaluative dimension needs to be changed in the light of new 
information). The second (and most important for a personalized decision 
aid) function is to create or change the preference models by means of 
which the options in the database are to be evaluated. It is this function 
that can most significantly be tailored to individual cognitive styles. 

The adjust module allows the user to specify preferences in three basically 
different ways: (1) compensatory knowledge about the relative importance 
(and tradeoffs) among evaluative criteria; (2) cutoff levels, i.e., non¬ 
compensatory goals, on attributes such that candidates not meeting the 
cutoffs are rejected; and (3) "bootstrapping" or "policy capture" judgments 
which express the values of options directly. These three modes correspond 
to differences in the degree to which users prefer analytic versus intui¬ 
tive and concrete approaches to choice. In addition, as noted, analytic 
knowledge in mode (1) may be expressed to virtually any degree of 
precision/imprecision or completeness/incompleteness. 

4.2.3 Decide module . The function of this module is to apply a user- 
preferred decision strategy, and a user-defined preference model to the 
database of options, and evaluate and display the results. If a full mul¬ 
tiattribute utility model has been specified (i.e., a complete set of 




numerical weights on all attributes), the system can display the most 
preferred alternative(s). If the model is only partially specified (i.e., 
user inputs of orderings, intervals and/or ratios of weights fall short of 
entailing exact values of weights), the system displays those implications 
that can be drawn from the information the system has. For example, the 
system can compute the feasible alternatives, i.e., those options that can¬ 
not be eliminated based on the available knowledge of the user's 
preferences. In addition, the system can perform a type of dominance 
testing; that is, given one alternative (hypothetical or from the 
database), it can compute which alternatives are known to be at least as 
good. Finally, the system can perform elimination by aspects, a choice 
strategy requiring only an ordering on the attributes and cutoff levels for 
each attribute. 

4.2.4 Select module . This module allows the user to select: (1) aspects 
of the preference model for display or adjustment, (2) information about 
options for display or adjustment, and (3) a subset of the current 
preference model for application in choice. 

4.2.5 Alert module . The function of this module is to notify the user of 
relevant changes in knowledge about options. This module is of primary im¬ 
portance in real-time systems in which the system must assimilate informa¬ 
tion about the outside world faster than it can be attended to by the user. 
Such a system must help the user to cope with "information overload" by 
alerting the user to possibly useful new information. 

Even in the present context, however, information overload may plague the 
user of a large database. If new options are incorporated into the 
database, or changes occur in the data regarding old options, then the user 
might be alerted if (and only if) those changes have implications for the 
current "best alternative set." 

A second mode of alerting involves user-specified alerts: for example, the 
system can be told to provide an alert if a candidate shows up with four 
years of programming experience. 










4.2.6 Advisory module . This module functions as an overseer of the other 
modules, checking when a preferred user decision strategy may result in 
significantly suboptimal choices. (1) Adjust: For example, users would be 
prompted when inputs provided under the Adjust module are significantly 
inconsistent, and ways of resolving the inconsistency would be suggested. 

If the information provided by a user is too imprecise or incomplete for 
successful application of the preferred decision strategy, Advisory prompts 
would suggest additional inputs that are needed (e.g., the number of live 
options could be narrowed down from 100 to the desired 10, if only the or¬ 
der of importance of education and experience is indicated). Users might 
also be notified if small changes in user inputs would have a significant 
impact on choice. (2) Decide: in the Decide module, users applying 
elimination-by-aspects or satisficing would be prompted to reconsider an 
option which has been discarded because it fails to meet a cutoff on one 
attribute, but is outstanding in other respects. Conversely, the user 
would be notified if an option which is accepted, because it clears all 
cutoffs, is in fact dominated (i.e., there is an option set which is at 
least as good as that option on all dimensions and better on at least one 
dimension). (3) Select: Users would be prompted when user-selected at¬ 
tributes fail to capture significant variance in the option database. (4) 
Alert: Users would be prompted when user-defined alerting criteria fail to 

capture significant aspects of the current preference model. 

In all cases, users would control the degree of significant of the problem 
that is to trigger an advisory prompt. Moreover, users are free to accept 
or reject any advice that is offered. 

4.3 Current Status 

This section describes in somewhat more detail selected functions in the 
personalized evaluation system that are currently implemented, partially 
implemented, and planned. 







4.3.1 Adjust . 


4.3.1.1 Problem structuring (partially implemented) --This function will 
allow initial structuring of the problem, i.e., identification of the op¬ 
tions to be considered and the attributes on which they are to be 
evaluated. The user will be able to organize the attributes hierarchically 
using interactive graphics for "tree-building," so that analyses performed 
at any level will be aggregated upward (or decomposed downward, if desired). 

4.3.1.2 Entering compensatorv/tradeoff information on attribute weights 
(partially implemented) --The user can enter compensatory/tradeoff informa¬ 
tion on the relative importance of attribute weights in any one of three 
ways (or in any combination of the three): 


• The Weights Screen (Implemented) - On the Weights screen 
(Figure 3), the user can enter upper and lower bounds for each 
attribute weight. By setting the upper and lower bounds equal, 
a precise weight may be entered. The user also sees the bounds 
implied by all other judgments (s)he has made on this and any 
other screens. Figure 3 shows a sample Weights screen display, 
on which the user has indicated, for example, that the weight 
for the first attribute (Alpha) should be between 35 and 60 on 
a scale from 0 to 100, with the other attributes weighted as 
shown. 

• The Ratio Screen (Implemented) - The Ratio screen (Figure 4) 
allows comparisons of relative magnitudes of attribute weights. 
One of the attributes may be specified as the standard, and 
other attribute weights are assessed relative to it. In the 
sample screen of Figure 4, the decision maker has indicated 
that the Beta attribute should be given weight between equal to 
and about 20% higher than that of the Alpha attribute. The in¬ 
formation is coded internally into linear inequality con¬ 
straints on the attributes. Again, the decision maker can see 
the implications of all other inputs to the system (on this or 
any other screen) as they relate to relative magnitudes of at¬ 
tribute weights. 

• The Order Screen (Partially Implemented) - On this screen the 
user can enter information merely about the order of importance 
of attribute weights (e.g., that the first attribute should 
have higher weight than the second without saying any more 
about what those weights are). Once again, the implications of 
all user inputs for rank order of weights is displayed. 
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Figure 3: Weights Screen 
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Figure 4: Ration Screen 






















Each of these screens allows a user to think about his or her preferences 
in a different way. Yet they are highly integrated, since the implications 
of all currently available information (entered from any of the above 
screens, or from the bootstrapping screen described below) can be displayed 
on each screen. The mechanism for computing implications across screens is 
linear programming. 

The entering of information on each of these screens corresponds to build¬ 
ing or changing a preference model. 

4.3.1.3 Cutoffs on attribute scores (implemented) --On the Cutoffs screen, 
the user can indicate minimal and/or maximal acceptable levels for any 
attribute, so that alternatives not meeting the cutoff are eliminated from 
consideration. Cutoffs can also be provided using the Elimination-by- 
Aspects function, described below. 

4.3.1.4 Bootstrapping (planned) --On the bootstrapping screen, the user 
makes direct judgments about alternatives, either real or hypothetical. 

The user may place bounds on the score of the alternative (e.g., "between 
65 and 80 on a scale of 100") or may make a direct comparison between two 
alternatives (e.g., "Jones would score better than Smith"). As for the 
above screens, bootstrapping information can be encoded as linear in¬ 
equalities on attribute weights. 

Bootstrapping is an indirect way of giving the system information on at¬ 
tribute weights. Hence, the Bootstrapping screen may be viewed as part of 
the Adjust module, and implications of the Bootstrapping judgments are 
stored as part of the Knowledge module. 

4.3.2 Decide (partially implemented) . Bv pointing to "Go" and clicking 
the mouse key, the user instructs the system to apply whatever knowledge 
about his preferences the system has gathered to the current set of 
options. Two settings are provided which influence the performance of this 
function: "# Needed" indicates the number of options which the user ul¬ 

timately desires to choose; it may range from 1 to the number of items in 
the database (a trivial choice problem!). The first time "Go" is selected, 









the preference model is applied to the entire database of options. The ap¬ 
plication of that model is likely to result in a smaller, weeded down sub¬ 
set of options which remains feasible or "live" (i.e., a set of options 
which is known to contain the desired choices, but within which further 
discriminations cannot be made based on present inputs). The user may, if 
he wishes, return to the Adjust screens, provide further inputs (or revise 
old ones), and then select "Go" again. Each subsequent selection of "Go" 
operates on the currently prevailing set of live options, further narrowing 
it down. The user has the option, however, of resetting this process by 
selecting "Use all." In that case, the next use of "Go" operates on the 
original, full database. 

By these simple commands, the user can implement any of a variety of choice 
strategies. For example, by setting cutoffs on additional dimensions be¬ 
tween each use of "Go," he may successively eliminate options, through an 
elimination-by-aspects strategy. If the user has provided information on 
cutoffs and on the ordering of dimensions, the system will in effect per¬ 
form an elimination-by-aspects analysis for him, applying the cutoffs on 
each dimension to the option set in the specified (or implied) order of 
importance, stopping if and when the option set is reduced to the desired 
number of choices. Any compensatory information will also be used directly 
by the system to eliminate options. The result of "Go" is always the set 
of feasible options, i.e., those that could be optional given current 
information. 

4.3.3 Guides. In addition to these basic functions, a variety of guides 
and special screens support the decision process. These involve functions 
not only from the Decide module, but also from Adjust, Select, and 
Advisory, orchestrated in a user-friendly dialogue that reflects typical 
decision making patterns. 

4.3.3.1 Eliminate (partially implemented) --For example, instead of per¬ 
forming elimination-by-aspects by means of elementary commands (as 
described above), the user can receive additional guidance through 
"Eliminate." This guide directs the user, in a highly flexible manner, 
through the following steps: 
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(1) Selection of attributes for inclusion in the choice process (by 
graphical pointing); 

(2) use of ordering screen to rank order the selected attributes, 
with prompts to elicit from the user a single connected 
ordering; 

(3) use of cutoff screen to elicit cutoffs for the selected 
attributes; 

(4) display of results on the EBA screen. 

These steps are by no means rigid. For example, if he or she is satisfied 
with the already existing attribute selection, ordering, and cutoff 
specification, the user may proceed directly to step (4). At any time 
thereafter, the user may return to any previous step, make changes, and ob¬ 
serve the results. 

The EBA screen (Figure 5) functions as an elimination-by-aspects 
"spreadsheet." It displays the selected attributes in order of importance, 
the lower and/or upper cutoffs, and the number of surviving options at each 
stage in the process (i.e., after each additional set of cutoffs has been 
applied). In addition, an advisory prompt is provided which notifies the 
user if there are promising options which have been rejected. These are 
options which fall outside the cutoffs on an attribute, but score very well 
on other dimensions. In addition, the user is told by how much the 
specified cutoffs would have to be stretched to include these rejected (but 
promising) options. 

The user can modify any cutoff directly on this screen, and the EBA will be 
recomputed, showing which alternatives are thereby excluded or re-included. 

By use of the mouse and the VIEW option (Figure 6), the user can "zoom" in 
on the details of any part of the EBA screen. For example, the user can 
view all database options in the vicinity of the lower or upper cutoff on 
any attribute, or he can view the set of promising but rejected options. 

The reasons for considering an option "promising" are also indicated. From 
the zoomed-in point, the user can scroll anywhere else in the database of 
options. The VIEW screen can also be used to modify cutoffs in a way that 
differs significantly from the CUTOFFS screen or the EBA screen. The user 
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may graphically move upper and lower cutoffs against the context of actual 
options to be included or excluded, rather than merely against an abstract 
numerical scale. 

4.3.3.2 Justify (partially implemented') --The Justify screen (Figure 7) 
provides users with a comprehensive evaluative overview of selected 
options. The objective is to support the final stages of a decision 
process, in which users seek to examine candidate choices in detail, to 
look for previously overlooked weak points, and to construct a justifica¬ 
tion for the choice. The Justify screen for a given option displays the 
score of that option on each attribute, the best score of any option on 
each attribute, and the difference (in standard deviations) between the 
current option's score and the best score; it also shows how many options 
scored better than the present one on each attribute and what percentage of 
the total database they represent. Finally, it indicates if the present 
option is poor or outstanding on any particular dimension. 

A VIEW option is associated with Justify which enables users to "zoom" in 
on and examine in detail the options which are superior to the given option 
in any particular dimension. 

For some decision makers, the Justify screen might play a central role in 
decision making. The process of justifying a choice may lead to recon¬ 
sideration and revision of previous preference judgments. Montgomery 
(1983) has argued that decision making in general is a search for good ar¬ 
guments or justifications. A characteristic strategy is to accept an op¬ 
tion as justified only if efforts to create a dominance structure for that 
option are successful, i.e., when preferences can plausibly be represented 
in such a way that the option appears as good or better than other options 
on all dimensions. A stronger justification occurs when the preferred op¬ 
tion is shown to be uniquely outstanding on at least one dimension. The 
Justify screen enables users to ascertain quickly whether requirements of 
this sort are satisfied. If not, users may alter the representation of 
their preference model until dominance applies, e.g., by combining at¬ 
tributes (via the problem structuring screen), by reducing the importance 
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of attributes on which the preferred option is poor, or by reconsidering 
the assessment of the performance of one or more options on an attribute. 

Advisory prompts will notify users when an option is itself dominated, or 
is especially poor in some dimension. In addition, the system will track 
user efforts to establish a dominance structure, and prompt when altera¬ 
tions in previous preferences are excessive according to a user-set 
criterion. 

4.3.3.3 Focus (planned) --Traditional multiattribute utility analysis 
requires precise numerical assessments of the relative importance of all 
evaluative dimensions. Both ELIMINATE and JUSTIFY simplify the choice 
process by enabling the user to employ simple non-compensatory judgments, 
i.e., comparisons within (rather than across) attributes. An ordering of 
attributes by importance is the only across-attribute information required. 
The result is a selection of options that may, on occasion, overlook impor¬ 
tant tradeoffs. Advisory prompts are provided to protect against this. An 
alternative strategy for simplifying choice, however, is to retain the goal 
of selecting alternatives with the highest expected utility (i.e., the 
highest overall score allowing for tradeoffs), but to elicit compensatory 
assessments from users only to the degree required by the particular choice 
problem at hand. 

The FOCUS guide steps the user through the minimal set of assessments 
required to narrow the option set to the number desired. FOCUS begins with 
the weakest possible test, dominance, to determine if that is sufficient to 
obtain the needed weeding out. Only if this fails does it step progres¬ 
sively through queries for stronger and stronger inputs from the user. In 
all cases, it seeks out aspects of preference knowledge that are likely to 
have the greatest impact on narrowing down the option set .a.g., the order 
of importance of attribute A and B; whether C is at least twice as impor¬ 
tant as D; etc.). As a result, the user's choice problem may be solved 
with the least judgment effort from the user. 

In addition, at any point, the user has the option of short-circuiting this 
process by instructing the system to "extrapolate" a set of constraints 
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that would solve the problem. The system then selects a set of weights 
consistent with the information thus far provided by the user, and ranks 
the alternatives with respect to those weights. A sensitivity analysis is 
also displayed, to permit the user to assess how sensitive the ranking Is 
to the set of weights chosen. If he desires, the user may then provide ad¬ 
ditional judgments of his own to "correct" the system's extrapolation. 


Implementation 


An implementation of the personnel selection aid is underway. This section 
provides a summary of the hardware and software approach being taken. 


4.4.1 Hardware . At present an IBM AT system is b°ing used for development 
of the aid. Software produced will operate on both PC- and AT-compatible 
machines. These are both 16-bit microprocessors, the PC having an 8-bit 
data path provided by the Intel 8088 processor, and the AT having a 16-bit 
data path due to its Intel 80286 processor. For development purposes, 
machines utilized will be equipped with floating point co-processors 
(8087/80287) to improve computational speed, hard disk.., and at least 512KB 
of random access memory. Software to be developed will not require the co¬ 
processors (but will utilize them when available) and will require no more 
than 256KB of memory (the minimum available on the PC or AT). 


The aiding system planned, like the submarine approach/attack planning aid 
developed under Phase I of this project, will utilize interactive graphics 
heavily. We anticipate that the present low-resolution (320 x 200 pixels 
with 4 simultaneous colors) video systems used by personal computer owners 
will be largely replaced within the next two to three years with higher 
resolution systems. For this reason, we are utilizing the recently avail¬ 
able IBM Enhanced Graphics Adapter (640 x 350 pixels with 16 simultaneous 
colors) and an RGB color monitor of no less than 640 x 350 pixel resolution 


We plan to continue using a mouse-based input device to minimize reliance 
on keyboard input, but will develop a parallel approach that will permit 
software operation with either mouse or keyboard. A light pen is supported 
as well. 
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4.4.2 Software Software will consist of the following: 

• A set of software modules written in C and comprising the user- 
system interface. These will provide the user link to the 
various personalizing modules (Select, Planning, Adjust, Alert, 
Advisory). These will be as generic as possible--suitable for 
rapid construction of personalized aids in other contexts. 

• A Data or Knowledge module will consist of two components: (1) 
a processing subsystem written in C containing the rules ap¬ 
propriate to support evaluations based on a variety of 
techniques, ranging from elimination by aspects to multiat¬ 
tribute utility analysis; (2) a database management subsystem, 
written in C and utilizing in addition components of an exist¬ 
ing system, dBASE III. 

• A graphics interface module. Because of the current lack of 
standards for graphics software and the variety of graphics 
hardware which must be supported if reasonable market penetra¬ 
tion is to be achieved, all graphics functions will be per¬ 
formed by a distinct software module. This will make future 
modifications relatively simple. This is being written in C. 

At present, we are utilizing a graphics system known as "Halo" 
and developed by Media Cybernetics, Inc. Programming of this 
system is somewhat similar to programming of the Virtual Device 
Interface (VDI) graphics system being released very shortly by 
IBM. As soon as feasible, we will replace the Halo system with 
the VDI system. The VDI approach offers the capability of 
automatically utilizing whatever resolution is available on a 
(VDI-supported) device. 
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5.0 CONCLUSION 


The work reported here has demonstrated that principles of personalized and 
prescriptive decision aiding, originally developed in the context of sub¬ 
marine command and control, can be generalized successfully to a quite dif¬ 
ferent decision making context. Submarine attack planning and personnel 
evaluation differ in degree of time stress, in the relative importance of 
inference and choice, in the availability of objective measures of success, 
in the organizational role of the aid and the user, and in numerous other 
ways. Yet each of these contexts poses a similar requirement for decision 
support that is tailored to individual styles of problem solving and deci¬ 
sion making and which provides, at the same time, prescriptive guidance and 
advice. A common set of cognitive interface modules has been found to 
satisfy this need in each case. 

Work to be done in the remainder of this project (Phase II) includes the 
following: 

• completed implementation of the prototype personalized evalua¬ 
tion system, as planned; 

• testing of the aid with potential users in the domain of per¬ 
sonnel selection (test materials involving a database of per¬ 
sonnel resumes have already been prepared); 

• demonstration of the system to ONR and other interested parties 

• exploration of Phase III support for technical refinement and 
testing of the aid, marketing and distrioucion, through con¬ 
tacts with commercial software firms and investment sources. 






REFERENCES 


Bariff, M.L., and Lusk, E.J. Cognitive and personality tests for the 
design of management information systems. Management Science, 1977, 23(8), 
820-829. 

Brown, R.V., Kahr, A.S., and Peterson, C.R. Decision analysis for the 
manager. NY: Holt, Rinehart and Winston, 1974. 

Chase, W.G., and Simon, H.A. The mind's eye in chess. In W.G. Chase 
(Ed.), Visual information processing. NY: Academic Press, 1973. 

Cohen, M.S. Decision support for attack submarine commanders: Attack 
planning and target range pooling (Technical Report 82-1). Falls Church, 
VA: Decision Science Consortium, Inc., April 1982. 

deFinetti, B. Foresight: Its logical laws, its subjective sources. 

English translation in H.E. Kybert, Jr., and H.E. Smokier (Eds.), Studies 
in subjective probability. NY: Wiley, 1964. (Original: 1937) 

Dawes, R.M. The mind, the model, and the task. In F. Restle et al. 

(Eds.), Cognitive theory. Hillsdale, NJ: Lawrence Erlbaum Assoc., 1975, 

1, 119-130. 

De Groot, A.D. Thought and choice in chess. The Hague: Mouton, 1965. 

Driver, M.J. and Mock, T.J. Human information processing, decision theory 
style, and accounting information systems. Accounting Review, 1976, 50, 
490-508. 

Edwards, W. The theory of decision making. Psychological Bulletin, 1954, 
54(4), 380-417. 

Edwards, W. Behavioral decision theory. Annual Review of Psychology, 

1961, 12, 473-498. 

Einhorn, H.J. Learning from experience and suboptimal rules in decision 
making. In T.S. Wallsten (Ed.), Cognitive processes in choice and decision 
behavior. Hillsdale, NJ: Lawrence Erlbaum Associates, Inc., 1980. 

Einhorn, H.J., and Hogarth, R.M. Behavioral decision theory: Processes of 
judgment and choice. Annual Review of Psychology, 1981, 32, 53-88. 

Einhorn, H.J., and Hogarth, R.M. Ambiguity and uncertainty in probabil¬ 
istic inference. Chicago, IL: University uf Chicago, Center for Decision 
Research, June 1984. 

French, S. On the methodology of decision analysis. I. What is a norma¬ 
tive theory of decision? Manchester: University of Manchester, Department 
of Decision Theory, March 1979. 

Hammond, K.R., Hamm, R.M., Grassia, J., and Pearson, T. The relative ef¬ 
ficacy of intuitive and analytical cognition: A second direct comparison 
(Report No. 252). Boulder, CO: University of Colorado, Center for Re¬ 
search on Judgment and Policy, June 1984. 


- 49 - 





Huber, G.P. Cognitive style as a basis for HIS and DSS designs: Much ado 
about nothing? Madison, WI: University of Wisconsin, 1982. 

Huysman, J.H. The effectiveness of the cognitive style construct in im¬ 
plementing operations research proposals. Management Science, 1970, 17, 
92-104. 

Johnson, E.J. Expertise and decision under uncertainty: Performance and 
process. Cambridge, MA: Massachusetts Institute of Technology, February 
25, 1985. 

Kadane, J.B., and Lichtenstein, S. A subjectivist view of calibration 
(Report 82-6). Eugene, OR: Decision Research, 1982. 

Kahneman, D., and Tversky, A. Subjective probability: A judgment of 
representativeness. Cognitive Psychology, 1972, 3, 430-454. 

Kahneman, D., and Tversky, A. Prospect theory: An analysis of decision 
under risk. Econometrics, March 1979, 47(2), 263-291. 

Kahneman, D., and Tversky, A. The psychology of preferences. Scientific 
American, January 1982, 246, 160-173. 

Keen, P.G.W. Adaptive design for decision support systems. Massachusetts 
Institute of Technology, Sloan School of Management, undated. 

Keeney, R.L., and Raiffa, H. Decisions with multiple objectives: 
Preferences and value tradeoffs. NY: Wiley and Sons, 1976. 

Klein, G.A. Automated aids for the proficient decision maker. IEEE Trans¬ 
actions on Systems, Man, and Cybernetics, 1980, 301-304. 

Leddo, J., Abelson, R.P., and Gross, P.H. Conjunctive explanations: When 
two reasons are better than one. Journal of Personality and Social 
Psychology, 1984, 47(5), 933-943. 

Libby, R., and Lewis, B.L. Human information processing research in 
accounting: The state of the art. Accounting , Organizations and Society, 
1977, 2(3), 245-268. 

Lichtenstein, S., Fischhoff, B., and Phillips, L.D. Calibration of 
probabilities: The state of the art to 1980. In D. Kahneman, P. Slovic, 
and A. Tversky (Eds.), Judgment under uncertainty: Heuristics and biases. 
NY: Cambridge University Press, 1982. 

Lopes, L.L. Procedural debiasing (WHIPP #15). Madison, WI: University of 
Wisconsin, Department of Psychology, October 1982. 

McKenney, J.L., and Keen, P.G.W. How managers' minds work. Harvard Busi¬ 
ness Review, May/June 1974, 52(3), 79-90. 

Mock, T.J., Estrin, T.L., and Vasarhelyi, M.A. Learning patterns, decision 
approach, and value of information. Journal of Accounting Research, 1972, 
129 151. 








Montgomery, H. Decision rules and the search for a dominance structure: 
Towards a process model of decision making. P. Humphreys, 0. Svenson and 
A. Vari (Eds.), Advances in Psychology, Amsterdam: North-Holland, 1983. 

Payne, J.W. Alternative approaches to decision-making under risk: Moments 
vs. risk dimensions. Psychological Bulletin , 1973, 80, 439-453. 

Payne, J.W. Task complexity and contingent processing in decision making: 
An information search and protocol analysis. Organizational Behavior and 
Human Performance , 1976, 16, 366-387. 

Payne, J.W. Information processing theory: Some concepts and methods ap¬ 
plied to decision research. In T.S. Wallston (Ed.), Cognitive processes in 
choice and decision behavior. Hillsdale, NJ: Lawrence Erlbaum Associates, 
Inc., 1981. 

Pitz, G.F., Sachs, N.J., and Heerboth, J. Procedures for eliciting choices 
in the analysis of individual decisions. Organizational Behavior and Human 
Performance, 1980, 26(3), 396-408. 

Raiffa, H. Decision analysis: Introductory lectures on choices under 
uncertainty. Reading, MA: Addison-Wesley, 1968. 

Russo, J.E., and Dosher, B.A. Cognitive effort and strategy selection in 
binary choice. University of Chicago, Graduate School of Business and 
Columbia University, May 1981. 

Sage, A.P. Behavioral and organizational considerations in the design of 
information systems and processes for planning and decision support. IEEE 
Transactions on Systems, Man, and Cybernetics, 1981, 11(9), 640-678. 

Savage, L.J. The foundations of statistics. NY: Wiley, 1954. 

Schroder, H.M., Driver, M.J., and Streufert, S. Human information 
processing. NY: Holt, Rinehart, and Winston, 1967. 

Schum, D., DuCharme, W., and DePitts, K. Research on human multistage 
probabilistic inference processes. Organizational Behavior and Human 
Performance, 1973, 10, 318-348. 

Schum, D.A., and Martin, A.W. Assessing the probative value of evidence in 
various inference structures (Research Report 81-02). Houston, TX: Rice 
University, 1981. 

Shaklee, H., and Fischhoff, B. Strategies of information search in causal 
analysis. Memory and Cognition, 1982. 

Simon, H.A. Models of man, social and rational: Mathematical essays on 
rational and human behavior in a social setting. NY: Wiley, 1957. 

Slovic, P., Fischhoff, B., and Lichtenstein, S. Accident probabilities and 
seat belt usage: A psychological perspective. Accident Analysis and 
Prevention, 1978, 10, 281-285. 












Streufert, S., and Streufert, S.C. Stress and the measurement of task 
performance. Decision making in complex tasks (Technical Report 3). 
Hershey, PA: The Milton S. Hershey Medical Center, Department of Be¬ 
havioral Science, 1981a. 

Streufert, S., and Streufert, S.C. Stress and information search in com¬ 
plex decision making: Effects of load and time urgency (Technical Report 
4). Hershey, PA: The Milton S. Hershey Medical Center, Department of Be¬ 
havioral Science, 1981b. 

Strub, M.H., and Levit, R.A. Computer compatibility with decision style: 
Eddies in a bit stream. In Proceedings of the Human Factors Society Annual 
Meeting, 1974. 

Svenson, 0. Process descriptions of decision making. Organizational Be¬ 
havior and Human Performance , 1979, 23, 86-112. 

Tversky, A. Elimination by aspects: A theory of choice. Psychological 
Review, 1972, 79(4), 281-299. 

Tversky, A., and Kahneman, D. Availability: A heuristic for judging 
frequency and probability. Cognitive Psychology, 1973, 4, 207-232. 

Tversky, A., and Kahneman, D. The framing of decisions and the psychology 
of choice. Science, 30 January 1981, 211, 453-458. 

Tversky, A., and Kahneman, D. Extensional vs. intuitive reasoning: The 
conjunction fallacy in probability judgment. Stanford, CA: Stanford 
University, Department of Psychology, June 1983. 

von Neumann, J., and Morgenstern, 0. Theory of games and economic behavior 
(2nd Ed.). Princeton, NJ: Princeton University Press, 1947. 

Wason, P.C. On the failure to eliminate hypotheses in a conceptual task. 
Quarterly Journal of Experimental Psychology , 1960, 12, 129-140. 

Wason, P.C. On the failure to eliminate hypotheses...A second look. In 
P.C. Wason and P.N. Johnson-Laird (Eds.), Thinking and reasoning, 1981, 7, 
242-248. 

Watson, S.R. Decision analysis as a replacement for cost-benefit analysis. 
European Journal of Operational Research, 1981, 7, 242-248. 

Wright, P., and Barbour, F. Phased decision strategies: Sequels to an 
initial screening. In M.K. Starr and M. Zeleny (Eds.), Multiple criteria 
decision making. Amsterdam: North-Holland, 1977. 

Zmud, R.W. Individual differences and MIS success: A review of the 
empirical literature. Management Science, October 1979, 25(10), 966-979. 












DISTRIBUTION LIST 


OSD 

CAPT Paul R. Chatelier 
Office of the Deputy Under Secretary 
of Defense 
OUSDRE (E&LS) 

Pentagon, Room 3D129 
Washington, D.C. 20301 

Department of the Naw 

Engineering Psychology Program 
Offie of Naval Research 
Code 1142EP 

800 North Quincy Street 
Arlington, VA 22217-5000 (3 copies) 

CAPT P.M. Curran 
Code 125 

Office of Naval Research 
800 North Quincy Street 
Arlington, VA 22217-5000 

Information Sciences Division 
Code 1133 

Office of Naval Research 
800 North Quincy Street 
Arlington, VA 22217-5000 

Special Assistant for Marine 
Corps Matters 
Code 00MC 

Office of Naval Research 
800 North Quincy Street 
Arlington, VA 22217-5000 

Director 

Technical Information Division 
Code 2627 

Naval Research Laboratory 
Washington, D.C. 20375-5000 

Dr. Neil McAlister 

Office of Chief of Naval Operations 

Command and Control 

OP-094H 

Washington, D.C. 20350 


Human Factors Department 
Code N-71 | 

Naval Training Systems Center j 

Orlando, FL 32813 j 

; 

Dr. A.L. Slafkosky j 

Scientific Advisor 
Commandant of the Marine Corps 
Washington, D.C. 20380 ! 

i 

i 

CDR C. Hutchins 1 

Code 55 j 

Naval Postgraduate School I 

Monterey, CA 93940 ! 

j 

Dr. Stanley Collyer 

Office of Naval Technology i 

Code 222 I 

800 North Quincy Street 1 

Arlington, VA 22217-5000 

Mr. Philip Andrews 
Naval Sea Systems Command 
NAVSEA 61R 

Washington, D.C. 20362 

Dr. George Moeller 
Human Factors Engineering Branch 
Naval Submarine Base 
Submarine Medical Research Lab. 

Groton, CT 06340 

Dr. Robert Blanchard 
Code 71 

Navy Personnel Research and 
Development Center 
San Diego, CA 92152-6800 

Dean of the Academic Departments 
U.S. Naval Academy 
Annapolis, MD 21402 

CDR W. Moroney 

Naval Air Development Center 
Code 602 

Warminster, PA 18974 







Department of the Army 

Dr. Edgar M. Johnson 
Technical Director 
U.S. Army Research Institute 
Alexandria, VA 22333-5600 

Technical Director 

U.S. Army Human Engineering Laboratory 
Aberdeen Proving Ground, MD 21005 

Department of the Air Force 

Dr. Kenneth R. Boff 
AF AMRL/HE 

Wright-Patterson AFB, OH 45433 

Dr. Earl Alluisi 
Chief Scientist 
AFHRL/CCN 

Brooks Air Force Base, TX 78235 

Other Government Agencies 

Defense Technical Information Center 
Cameron Station, Bldg. 5 
Alexandria, VA 22314 (2 copies) 

Dr. Clinton Kelly 

Defense Advanced Research Projects Agency 
1400 Wilson Blvd. 

Arlington, VA 22209 


Other Organizations 

Dr. Jesse Orlansky 
Institute for Defense Analyses 
1801 N. Beauregard Street 
Alexandria, VA 22311 

Dr. Stanley Deutsch 
NAS-National Research Council 
(COHF) 

2101 Constitution Ave., N.W. 
Washington, D.C. 20418 

Dr. Richard Pew 

Bolt, Beranek & Newman, Inc. 

50 Moulton Street 
Cambridge, MA 02238 

Dr. Baruch Fischhoff 
Perceptronics, Inc. 

6271 Variel Ave. 

Woodland Hills, CA 91367 

Dr. Bruce Hamill 
The Johns Hopkins Univ. 
Applied Physics Lab 
Laurel, MD 20707 




























